The Self Organization of Context for Learning in Multiagent Games

Reinforcement learning is an effective machine learning paradigm in domains represented by compact and discrete state-action spaces. In highdimensional and continuous domains, tile coding with linear function approximation has been widely used to circumvent the curse of dimensionality, but it suffers from the drawback that human-guided identification of features is required to create effective tilings. The challenge is to find tilings that preserve the context necessary to evaluate the value of a state-action pair while limiting memory requirements. The technique presented in this paper addresses the difficulty of identifying context in high-dimensional domains. We have chosen RoboCup simulated soccer as a domain because its 90-dimensional continuous state space makes it a formidable challenge for reinforcement learning algorithms. Using self-organizing maps and reinforcement learning in a two-pass process, our technique scales to large state spaces without requiring a large amount of domain knowledge to automatically form abstractions over the state space. Results show that our algorithm learns to play the game of soccer better than contemporary learned and handcoded opponents.